[NV] Add gb300 nvfp4 qwen3.5 MTP 8k1k recipes #2218
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 如需进行 PR 验证,请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
There was a problem hiding this comment.
Not approving — the pr-link: TBD issue flagged inline is real and will hard-fail the standard merge_with_reuse.sh canonicalization step (verified against utils/validate_perf_changelog.py and utils/prepare_perf_changelog_merge.py), so this needs a fix before merge.
Extended reasoning...
Overview
This PR adds 9 new MTP (multi-token prediction) benchmark recipe YAMLs for Qwen3.5-397B-A17B-NVFP4 on GB300, plus a corresponding configs/nvidia-master.yaml entry, a perf-changelog.yaml entry, and a new elif branch in runners/launch_gb300-nv.sh to route this model/precision/framework combination to the correct srt-slurm checkout and recipe overlay directory.
Security risks
None. This is benchmark configuration and a shell-script routing branch for an internal CI runner; no user input, auth, or secrets handling is involved.
Level of scrutiny
This falls into the repo's standard recipe-addition pattern (new YAML configs + master-config entry + changelog + runner routing), which is normally simple and mechanical. However, I independently verified the inline bug report's core claim by reading utils/validate_perf_changelog.py and utils/prepare_perf_changelog_merge.py: PR_LINK_PLACEHOLDERS is exactly {"XXX", ".../pull/XXX"}, and validate_added_pr_link/canonicalize_appended_links raise ChangelogValidationError for any pr-link value that isn't the canonical URL or one of those placeholders. The new changelog entry uses the literal string TBD, which matches neither, so the standard merge_with_reuse.sh path used by the merge-prs skill would hard-fail on this PR. That's a legitimate blocker for the standard merge tooling, even though it doesn't show up in PR-time CI checks (which don't invoke this validator).
Other factors
This is a one-line, easy fix (swap TBD for XXX or the real PR link), and the nit about mislabeling the recipes as "STP" instead of "MTP" in the changelog description is cosmetic. Neither issue affects the correctness of the recipe YAMLs, master-config entries, or runner routing logic themselves. Given the merge-blocking nature of the pr-link issue, I'm deferring rather than approving so a human/author can land the fix.
| - "Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18 with Dynamo installed at runtime (version 1.3.0.dev20260708, TCP request plane)." | ||
| - "9 STP configs: 1P1D DEP2+1D TEP16 (conc 48), 1P4D DEP2+4D TP16 (conc 24), 1P7D DEP2+7D TEP8 (conc 42), 2P15D DEP2+15D TP4 (conc 90), 3P4D DEP2+4D TEP8 (conc 192), 3P7D DEP2+7D TEP8 (conc 168), 4P1D DEP2+1D DEP16 (conc 384), 6P5D DEP2+5D TEP8 (conc 480), 9P1D DEP2+1D DEP16 (conc 1536)." | ||
| - "Runner: launch_gb300-nv.sh updated to clone NVIDIA/srt-slurm@sa-submission-q2-2026 and overlay local recipes from benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/ for the dynamo-trt+qwen3.5+fp4 path; --no-preflight added for model on compute-node /scratch." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 The new perf-changelog.yaml entry (line 4792) sets pr-link: TBD, but the standard merge tooling (utils/merge_with_reuse.sh → prepare_perf_changelog_merge.py canonicalize) only accepts the canonical .../pull/2218 URL or the literal XXX placeholder — TBD matches neither and will raise ChangelogValidationError, hard-failing the merge step even though PR-time checks stay green. Fix by changing it to https://github.com/SemiAnalysisAI/InferenceX/pull/2218 (or XXX, which the tooling auto-replaces at merge time).
Extended reasoning...
What the bug is: Every other entry in perf-changelog.yaml fills pr-link with either a concrete https://github.com/SemiAnalysisAI/InferenceX/pull/<n> URL or the sanctioned XXX placeholder that merge tooling auto-fills. This PR's new entry (appended at line 4792) instead uses the literal string TBD, which is not a value the changelog tooling recognizes.
Where it breaks: utils/validate_perf_changelog.py defines PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}. Its validate_added_pr_link(link, pr_number) function checks: if a PR number is known, the link must equal the expected canonical URL or be one of PR_LINK_PLACEHOLDERS — otherwise it raises ChangelogValidationError("new PR entry must use '<url>' or an XXX placeholder; found '<link>'"). TBD satisfies neither condition.
Why PR-time checks don't catch it: The sweep-gate path (run-sweep.yml → validate_perf_changelog.py main → process_changelog.py) does not call validate_added_pr_link at all — and separately, ChangelogEntry.pr_link in matrix_logic/validation.py is an unconstrained str, so any string passes Pydantic validation. This is exactly why the issue is invisible in the PR's green checks.
Where it does fire: utils/merge_with_reuse.sh, the repository's standard reuse-assisted merge path (used by the merge-prs skill), unconditionally runs prepare_perf_changelog_merge.py canonicalize --pr-number <PR> after merging main (line 181), regardless of whether a conflict occurred. canonicalize_appended_links() calls compare_entries(base, head, pr_number), which for every appended entry calls validate_added_pr_link(entry['pr-link'], pr_number). With pr_number=2218 and link='TBD', this raises ChangelogValidationError: new PR entry must use 'https://github.com/SemiAnalysisAI/InferenceX/pull/2218' or an XXX placeholder; found 'TBD'.
Step-by-step proof:
- PR [NV] Add gb300 nvfp4 qwen3.5 MTP 8k1k recipes #2218 is merged/rebased via
merge_with_reuse.sh. - After the main-merge step, the script calls
prepare_perf_changelog_merge.py canonicalize --changelog-file perf-changelog.yaml --base-ref origin/main --pr-number 2218 --repo SemiAnalysisAI/InferenceX. canonicalize_appended_linksparses base vs. head, finds one appended entry (qwen3.5-fp4-gb300-dynamo-trt-mtp), and callsvalidate_added_pr_link('TBD', 2218).expected = 'https://github.com/SemiAnalysisAI/InferenceX/pull/2218';'TBD' not in PR_LINK_PLACEHOLDERSand'TBD' != expected→ChangelogValidationErroris raised.merge_with_reuse.shexits non-zero at that step, blocking the automated merge until someone manually fixes the changelog entry.
Impact and fix: Had the author used the sanctioned XXX placeholder, canonicalize_appended_links would have auto-rewritten it to the real /pull/2218 link and the merge would proceed silently. Using TBD instead turns what should be a no-op auto-fix into a hard failure of the standard merge tooling. The fix is a one-line change: replace TBD with either XXX or the literal https://github.com/SemiAnalysisAI/InferenceX/pull/2218.
|
|
||
| - config-keys: | ||
| - qwen3.5-fp4-gb300-dynamo-trt-mtp | ||
| description: | ||
| - "Add Qwen3.5-397B-A17B-NVFP4 GB300 disaggregated TensorRT-LLM+Dynamo MTP (3-token) benchmarks for 8k1k." | ||
| - "Image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18 with Dynamo installed at runtime (version 1.3.0.dev20260708, TCP request plane)." | ||
| - "9 STP configs: 1P1D DEP2+1D TEP16 (conc 48), 1P4D DEP2+4D TP16 (conc 24), 1P7D DEP2+7D TEP8 (conc 42), 2P15D DEP2+15D TP4 (conc 90), 3P4D DEP2+4D TEP8 (conc 192), 3P7D DEP2+7D TEP8 (conc 168), 4P1D DEP2+1D DEP16 (conc 384), 6P5D DEP2+5D TEP8 (conc 480), 9P1D DEP2+1D DEP16 (conc 1536)." | ||
| - "Runner: launch_gb300-nv.sh updated to clone NVIDIA/srt-slurm@sa-submission-q2-2026 and overlay local recipes from benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/ for the dynamo-trt+qwen3.5+fp4 path; --no-preflight added for model on compute-node /scratch." | ||
| pr-link: TBD |
There was a problem hiding this comment.
🟡 The new perf-changelog.yaml entry for qwen3.5-fp4-gb300-dynamo-trt-mtp mislabels the 9 new recipes as "9 STP configs" (line ~4790), but this PR adds MTP (multi-token prediction) recipes, not STP — likely a copy-paste leftover from the sibling STP PR #2206. Should read "9 MTP configs" for accuracy.
Extended reasoning...
The new perf-changelog.yaml entry for qwen3.5-fp4-gb300-dynamo-trt-mtp is internally self-contradictory. The first description bullet correctly states "Add Qwen3.5-397B-A17B-NVFP4 GB300 disaggregated TensorRT-LLM+Dynamo MTP (3-token) benchmarks for 8k1k," but the third bullet begins "9 STP configs: 1P1D DEP2+1D TEP16 (conc 48), ...". STP (single-token prediction, i.e. no speculative decoding) and MTP (multi-token prediction, 3 speculative draft tokens) are different decoding modes, and this entry unambiguously documents the latter.
Every piece of evidence in this PR confirms these are MTP recipes: the master-config key is qwen3.5-fp4-gb300-dynamo-trt-mtp with every search-space point set to spec-decoding: "mtp"; all 9 new recipe filenames under benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/mtp/ contain mtp3; and every recipe's speculative_config block sets decoding_type: MTP with max_draft_len: 3. There is no STP recipe anywhere in this diff.
The likely root cause is that this changelog entry was drafted by copying the changelog entry from the sibling STP recipes PR (#2206), which does add STP configs, and the "STP" wording in the count line was never updated to match the MTP content being described here — while the first bullet was correctly edited to say "MTP."
Step-by-step proof:
- Read
perf-changelog.yamllines 4784-4792 (new entry): bullet 1 says "...MTP (3-token) benchmarks..."; bullet 3 says "9 STP configs: ...". - Read
configs/nvidia-master.yaml, new keyqwen3.5-fp4-gb300-dynamo-trt-mtp: every one of the 9search-spaceentries setsspec-decoding: "mtp". - Read any of the 9 new recipe YAMLs, e.g.
ctx1dep2_gen1tep16_mtp3_c48.yaml: bothprefill.speculative_config.decoding_typeanddecode.speculative_config.decoding_typeareMTPwithmax_draft_len: 3. - Filenames themselves encode
mtp3(3-token MTP), notstp. - Therefore "STP" in bullet 3 is factually wrong for the content it describes and contradicts bullet 1 of the same entry.
This is purely a documentation string in a changelog file with no functional/runtime impact — it doesn't affect recipe execution, CI, or the master config — so it does not block merge. The fix is a one-word edit: change "9 STP configs" to "9 MTP configs" on line 4790.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29378593024 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29391306858 |
Add Qwen3.5-397B-A17B-NVFP4 GB300 disaggregated TensorRT-LLM + Dynamo MTP benchmarks (8k1k)
Summary
This PR adds multi-token prediction (MTP, 3 speculative tokens) benchmark recipes for Qwen3.5-397B-A17B-NVFP4 on GB300 in a disaggregated TensorRT-LLM + Dynamo setup, covering the 8k1k (ISL 8192 / OSL 1024) scenario.
Runner and master config changes follow the same pattern as PR #2206 (STP recipes), which must be merged (or the runner branch must be compatible) before CI jobs launched from this PR will succeed.
Changes
9 MTP recipe configs —
benchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/8k1k/mtp/All configs use:
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc181.3.0.dev20260708, TCP request planeqwen3.5-fp4→/scratch/models/Qwen3.5-397B-A17B-NVFP4DEP2(2 GPUs, 1 node); decode topologies varymtp3in filename)ctx1dep2_gen1tep16_mtp3_c48ctx1dep2_gen4tp16_mtp3_c4ctx1dep2_gen7tep8_mtp3_c42ctx2dep2_gen15tp4_mtp3_c90ctx3dep2_gen4tep8_mtp3_c192ctx3dep2_gen7tep8_mtp3_c168ctx4dep2_gen1dep16_mtp3_c384ctx6dep2_gen5tep8_mtp3_c480ctx9dep2_gen1dep16_mtp3_c1536runners/launch_gb300-nv.shelifbranch fordynamo-trt + qwen3.5 + fp4: clonesNVIDIA/srt-slurm@sa-submission-q2-2026and overlaysbenchmarks/multi_node/srt-slurm-recipes/trtllm/qwen3.5/gb300-fp4/disagg/(covers bothstp/andmtp/subdirs)--no-preflightfor this path — model lives on compute-node/scratch/NVMe, not visible from the GHA runner podconfigs/nvidia-master.yamlqwen3.5-fp4-gb300-dynamo-trt-mtp: 9 search-space points, allspec-decoding: "mtp",disagg: true,kv-p2p-transfer: nixlperf-changelog.yamlqwen3.5-fp4-gb300-dynamo-trt-mtp